Telegram Group & Telegram Channel
مقایسه زمانی BPE Tokenizer روی دو کتابخونه Hugging Face Tokenizers و OpenAI TikToken روی ولیدیشن دیتاست تاینی‌استوریز:

dataset = load_dataset("roneneldan/TinyStories")
texts = dataset["validation"]["text"]

# Load the GPT-2 tokenizer for both libraries
tiktokenizer = tiktoken.get_encoding("gpt2") # tiktoken
hf_tokenizer = Tokenizer.from_pretrained("gpt2") # Hugging Face tokenizers

# Measure tiktoken speed
start_time = time.time()
tiktoken_results = [tiktokenizer.encode(text) for text in texts]
tiktoken_time = time.time() - start_time

# Measure tokenizers speed
start_time = time.time()
hf_results = [hf_tokenizer.encode(text).ids for text in texts]
hf_time = time.time() - start_time

# Print results
print(f"tiktoken Time: {tiktoken_time:.4f} seconds")
print(f"tokenizers Time: {hf_time:.4f} seconds")

tiktoken Time: 2.6481 seconds
tokenizers Time: 16.7744 seconds



tg-me.com/pytorch_howsam/671
Create:
Last Update:

مقایسه زمانی BPE Tokenizer روی دو کتابخونه Hugging Face Tokenizers و OpenAI TikToken روی ولیدیشن دیتاست تاینی‌استوریز:

dataset = load_dataset("roneneldan/TinyStories")
texts = dataset["validation"]["text"]

# Load the GPT-2 tokenizer for both libraries
tiktokenizer = tiktoken.get_encoding("gpt2") # tiktoken
hf_tokenizer = Tokenizer.from_pretrained("gpt2") # Hugging Face tokenizers

# Measure tiktoken speed
start_time = time.time()
tiktoken_results = [tiktokenizer.encode(text) for text in texts]
tiktoken_time = time.time() - start_time

# Measure tokenizers speed
start_time = time.time()
hf_results = [hf_tokenizer.encode(text).ids for text in texts]
hf_time = time.time() - start_time

# Print results
print(f"tiktoken Time: {tiktoken_time:.4f} seconds")
print(f"tokenizers Time: {hf_time:.4f} seconds")

tiktoken Time: 2.6481 seconds
tokenizers Time: 16.7744 seconds

BY PyTorch Howsam


Warning: Undefined variable $i in /var/www/tg-me/post.php on line 283

Share with your friend now:
tg-me.com/pytorch_howsam/671

View MORE
Open in Telegram


PyTorch Howsam Telegram | DID YOU KNOW?

Date: |

How to Use Bitcoin?

n the U.S. people generally use Bitcoin as an alternative investment, helping diversify a portfolio apart from stocks and bonds. You can also use Bitcoin to make purchases, but the number of vendors that accept the cryptocurrency is still limited. Big companies that accept Bitcoin include Overstock, AT&T and Twitch. You may also find that some small local retailers or certain websites take Bitcoin, but you’ll have to do some digging. That said, PayPal has announced that it will enable cryptocurrency as a funding source for purchases this year, financing purchases by automatically converting crypto holdings to fiat currency for users. “They have 346 million users and they’re connected to 26 million merchants,” says Spencer Montgomery, founder of Uinta Crypto Consulting. “It’s huge.”

NEWS: Telegram supports Facetime video calls NOW!

Secure video calling is in high demand. As an alternative to Zoom, many people are using end-to-end encrypted apps such as WhatsApp, FaceTime or Signal to speak to friends and family face-to-face since coronavirus lockdowns started to take place across the world. There’s another option—secure communications app Telegram just added video calling to its feature set, available on both iOS and Android. The new feature is also super secure—like Signal and WhatsApp and unlike Zoom (yet), video calls will be end-to-end encrypted.

PyTorch Howsam from ru


Telegram PyTorch Howsam
FROM USA